Papers with causal language modeling
GRITHopper: Decomposition-Free Multi-Hop Dense Retrieval (2026.eacl-long)
Copied to clipboard
| Challenge: | Decomposition-based multi-hop retrieval methods rely on autoregressive steps to break down complex queries, which breaks end-to-end differentiability and is computationally expensive. |
| Approach: | They propose a multi-hop dense retrieval model that integrates causal language modeling with dense retrievals. |
| Outcome: | The proposed model outperforms existing methods on in-distribution and out-of-difference benchmarks. |
Lexical Substitution as Causal Language Modeling (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods for lexical substitution task lacks autoregressive decoding capabilities. |
| Approach: | They propose a framework that uses causal language modeling (CLM) for lexical substitution task. |
| Outcome: | The proposed system outperforms GeneSis, the best previously published supervised LST method. |
Plug and Play Knowledge Distillation for kNN-LM with External Logits (2022.aacl-short)
Copied to clipboard
| Challenge: | Despite the promising evaluation results by knowledge distillation (KD) in natural language understanding (NLU) and sequence-to-sequence (seq2sequ) tasks, KD for causal language modeling (LM) remains a challenge. |
| Approach: | They propose to use external logits to improve a student's kNN-LM by leveraging teacher's knowledge at test time. |
| Outcome: | The proposed method improves a student's kNN-LM in multiple language modeling datasets and improves perplexity. |
Large Product Key Memory for Pretrained Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing product key memory (PKM) models that increase model capacity with insignificant computational overhead are limited to causal language modeling. |
| Approach: | They propose product key memory (PKM) that enables very efficient and exact nearest neighbor search in a large number of learnable memory slots. |
| Outcome: | The proposed product key memory improves model capacity and performance by replacing a feed-forward network with a model weighted model. |
Confounding Factors in Relating Model Performance to Morphology (2025.emnlp-main)
Copied to clipboard
| Challenge: | morphological differences between languages are unclear, but are often considered unimportant . confounding factors make it hard to compare results and draw conclusions, authors argue . |
| Approach: | They propose to use token bigram metrics to predict difficulty of causal language modeling . they argue that confounding factors are contributing to the conflicting evidence . |
| Outcome: | The proposed metrics better capture the relation between morphology and tokenization compared to word-based models. |
A Closer Look at Parameter Contributions When Training Neural Language and Translation Models (2022.coling-1)
Copied to clipboard
| Challenge: | Neural models and Transformers have been used for almost every NLP task . however, the intrinsic dynamics of the training procedure have not been studied in depth for highly complex network architectures. |
| Approach: | They analyze the learning dynamics of neural language and translation models using Loss Change Allocation indicator . they use a standard Transformer architecture to train a model with three learning objectives . |
| Outcome: | The proposed model is based on a standard model that is used for training tasks. |
DavIR: Data Selection via Implicit Reward for Large Language Models (2025.acl-long)
Copied to clipboard
Haotian Zhou, Tingkai Liu, Qianli Ma, Yufeng Zhang, Jianbo Yuan, Pengfei Liu, Yang You, Hongxia Yang
| Challenge: | 6% of Alpaca dataset selected with DavIR can steer both LLaMA and Gemma models to produce superior performance compared to the same models trained on the full 52K dataset. |
| Approach: | They propose a model-based data selection method for post-training Large Language Models . they generalize Reducible Holdout Loss to core-set selection problem of causal language modeling . |
| Outcome: | The proposed method can steer both LLaMA and Gemma models to superior performance compared to the same models trained on the full 52K dataset. |
Understanding Token Probability Encoding in Output Embeddings (2025.coling-main)
Copied to clipboard
| Challenge: | a common log-linear encoding of output token probabilities is used in language models, but it is sparse and inaccurate. |
| Approach: | They propose an approximate log-linear encoding of output token probabilities within the output embedding vectors and show that it is accurate and sparse. |
| Outcome: | The proposed output embeddings capture the corpus token frequency information in early steps, even before an obvious convergence of parameters starts. |
Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for token-level KV optimization and grouping of tokens are inefficient and strain compute and storage resources. |
| Approach: | They propose a mixture-of-expert approach that dynamically optimizes token-wise computation and memory allocation by a token-based expert-choice routing mechanism guided by learned importance scores. |
| Outcome: | The proposed approach retains all tokens while adaptively routing them to specialized experts with varying KV group sizes, balancing granularity and efficiency. |